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ABSTRACT 



A suggestion is made for improving the Feldman- Cousins method of estimating sig- 
nal counts in the presence of background. The method concentrates on finding essential 
information about the signal and ignoring extraneous information about background. An 
appropriate method is found which uses the condition that that the number of background 
events obtained does not exceed the total number of events obtained. Several alternative 
approaches are explored. 



1. Introduction 

Feldman and Cousins, 1 in a recent article, have made major advances towards solv- 
ing two long-standing problems concerning the use of confidence levels for estimating a 
parameter from data. The first of these is eliminating the bias that occurs when one 
decides between using a confidence interval or a confidence bound, after examining the 
data. The second is finding a confidence interval when the experimental result produces 
estimators that are close to or past known bounds for the parameters of interest. Feld- 
man and Cousins' method is called the unified approach below and is described in Section 
2. In the present paper we argue that the unified approach does not make quite enough 
of an allowance for the known bounds and suggest a modification. The modification is 
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illustrated with the KARMEN 2 Data , where precisely this problem has arisen. The 
KARMEN group has been searching for a neutrino oscillation signal reported by an LSND 
experiment. As of Summer 1998, they had expected to see 2.88 ± 0.13 background events 
and 1.0 - 1.5 signal events, if the LSND results were real, but had seen no events. From 
their analysis, they claimed to almost exclude the effect claimed by the LSND experiment. 

To be specific recall that the Poisson density with mean \i is 

PM = \/e-» (1) 

for k = 0, 1,2, and let P M denote the corresponding distribution function, P^k) = 
Pn(0) + ■ ■ ■ + Pfi(k). Suppose that background radiation is added to a signal producing 
a total observed count, n say, that follows a Poisson distribution with mean b + A. Here 
the background and signal are assumed to be independent Poisson random variables, with 
means b and A respectively. What are appropriate confidence intervals for A if no events 
are observed (n = 0) or, more generally, if n is smaller than 6? For n = and a 90% 
confidence level, the unified intervals all have left endpoints at A = 0, while the right 
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endpoints decrease from 2.44 when b = to 0.98 when 6 = 5. These are the right answers 
within the formulation of the unified approach. 

1)1 We use here the published numbers for n = given by Feldman and Cousins. The numbers we obtain 
differ slightly. For 6 = 3, n = 0, we obtain 0.95 and for b = 5, n = 0, we obtain 0.77. 
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The formulation is suspect, however, because the confidence intervals should not de- 
pend on b when n = 0. For if no events are observed, then both the signal and background 
radiation must have been zero. It is as if two independent experiments were performed, 
one for the background and one for the signal. The fact that there were no background 
events may be interesting but it is not directly relevant to inference about A once the signal 
is known, and certainly the a priori expectation b of the background radiation is irrelevant 
when one knows that the actual background was 0. In this case, the confidence interval 
for A should be the same as if one had observed a signal of strength 0-either 2.44 using 
the unified approach, or 2.30 using an upper confidence bound. Statisticians have a name 
situations like this one. The background radiation is called an ancillary variable, because 
its distribution does not depend on unknown parameters, and conventional statistical wis- 
dom calls for conditioning on ancillary variables when possible. 4 That is what we just did, 
since conditioning on no background events leaves n as the signal. 

Our modification is described in Section 2, where it is compared to the unmodified 
procedure. For the KARMEN 2 data the modified confidence region is substantially larger 
than the unmodified one and overlaps the major portion of the LSND region. The modi- 
fication is compared to a Bayesian solution in Section 4 and shown to agree with it quite 
well, especially for low counts. Some other possible modifications are discussed briefly in 
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Section 3. Giunti has also proposed a modification of the unified approach and applied 
it to the KARMEN 2 data. Our approach is contrasted with his in Section 3. 



2. An Improved Method 

It is not trivial to generalize the method just described to the case of non-zero counts 
n that may be small compared to the expected background radiation. For if n > 0, then 
it is no longer possible to recover the background and signal. The key to our modification 
is to remember that a confidence interval consists of values of the parameter that are 
consistent with the data (that is, are not rejected by an hypothesis test whose significance 
level is one minus the confidence level). This is also the approach taken by Feldman and 
Cousins. Suppose, for example, that the expected backgound radiation is b = 3 but that 
only one event is observed (n = 1). Is A = 2 inconsistent with this observation? From 
one point of view it is. If A = 2, then the probability of observing at most one event 
is e~ 5 + 5e~ 5 = 6e~ 5 = .040, which is less than the usual levels of significance. On the 
other hand, if only one event is observed, then there can have been at most one background 
event, and this information should be included in assessing significance. For the probability 
of at most one background event, e~ 3 + 3e~ 3 = 4e~ 3 = .199, is not large, and if the 
statement A = 2 is regarded as an hypothesis, then it seems unfair to include lower than 
expected background radiation as evidence against it. The way to remove the effect of the 
low background radiation is to compute the conditional probability of at most one event 
(total), given at most one background event. The latter is 6e~ 5 /4e~ 3 = 1.5 x e~ 2 = .203, 
which is not less than the usual levels of significance. 
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Some notation is required to adapt this reasoning to the unified approach. The like- 
lihood function in the signal plus background problem is L&(A|n) = Pb+\{n), where n is 
the observed count. Following Feldman and Cousins, let A = max[0, n — b] denote the 
maximum likelihood estimator of A and let 

Rb(x , n) =bm (2) 

L b (X\n) 

be the likelihood ratio statistic for testing A. Then the unified approach consists of taking 
those A for which i?(A, n) > c(A), where c(A) is the largest value of c for which 

E Pb+\( k ) < a ( 3 ) 

k:R b (\,k)<c 

and 1 — a is the desired confidence level. In words, the left side of (3) is the probability 
that Rf,(X,n) < c; a level a generalized likelihood ratio test 6 rejects the hypothesis A = Ao 
if Rb(\o,n) < c(Ao); and the unified confidence intervals consist of those A that are not 
rejected. The modification suggested here consists of replacing pb+\(k) by the conditional 
probability of exactly k events total given at most n background events. The latter is 

q n _ | Pb+x(k)/Pb{n) if k < n 

\YTj=QPb{j)p\{k - j) / Pb{n) if/.- > //. 

since k total events imply at most n background events when k < n. Let -R^(A, k) denote 
the likelihood ratio obtained using q^ x (k); z.e., -R^ (A, k) = q£ x (k)/~ma,x\>q'g x ,(k). Let c n (A) 
be the largest value of c for which 

E < «• (5) 

k:Fq{\k)<c 

Then the modified confidence interval consists of those A for which R^(X,n) > c n (A). 

The modified and original unified approaches are compared in Figure 1 for the special 
case 6 = 3 and n = 0, • • • , 15. Observe that the modified intervals are wider for small n and 
that there is not much difference for large n. The latter is to be expected, since there is 
not much difference between q£ x and Pb+\ for large n. In the case of small n, the rationale 
for the modification is as above. If n is smaller than 6, then there was less background 
radiation than expected, and this information should be used in assessing significance. 

For the KARMEN 2 Data, b = 2.88 ±0.13 and n = 0. At the 90% confidence level, the 
unified approach leads to < A < 1.08, and the modified interval leads to < A < 2.42. 
As above, values of A between 1.08 and 2.42 are found to be inconsistent with the data by 
the unified approach, but this is due to lower than expected background radiation, and the 
inconsistency disappears after adjusting for the low background radiation. On the basis of 
this data, it is not reasonable to exclude the possibility of signal. 
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Figure 1. The 90% C.L. region for an unknown Poison signal A in the presence of 
a Poisson background 6 = 3. The dashed lines and solid lines correspond to the unified 
approach and the modified approach, respectively. 

To be complete (and fair) Feldman and Cousins were aware of the problem with small 
counts. For such cases, they suggested reporting the the average upper limit that would be 
obtained by an ensemble of experiments with the expected background and no true signal, 
along with the intervals for the observed n. The conceptual difference between our intervals 
and the unified method is that our confidence levels are conditional and, therefore, refer 
to a different ensemble. In general terms, the main reason for conditioning is to obtain a 
model that describes the experiment performed more accurately. The price paid for the 
more accurate model is often a loss of power, or longer confidence intervals, and the effect 
can be large, as in the KARMEN data. Of course, power is important, but it is an illusion 
if the model does not describe the experiment well. 

The reader may be familiar with conditioning in the context of contingency tables 
when some of the row and/or column totals are fixed, have known distributions, or have 
distributions that only depend on nuisance parameters. In such cases it is appropriate 
to condition on the known totals, and this affects the distribution of tests statistics and 
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estimators. Fisher's exact test provides a specific example. See Lehmann for a derivation 

g 

of the exact test and Berkson for criticisms. Other reasons for conditioning arise when the 
precision with which an experiment was done is observed as part of the outcome. In the 
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case n = 0, our use of conditioning is consistent with these precedents. In the case n > 0, 
however, our use of conditioning goes beyond these established precedents because we 
condition on an observed bound for an ancillary variable, not the exact value. Our reasons 
for conditioning, illustrated by the numerical example with n = 1 above, are consistent 
with the precedents. To summarize these reasons: it seems unwise to regard lower than 
expected background radiation as evidence against a value of A. 



3. Other Possible Modifications 

The rationale given for the modification in Section 2 could also have been used to 
support other modifications. We describe these briefly here and explain our preference for 
the one described in Section 2. We also contrast our modification with that of Giunti. 

The modification described in Section 2 replaces Pb+\ with q£ x in the derivation of 
the unified approach, thus replacing Rb(X,k) by R^(X,k) = q£ x (k) / m&xy q£ x ,(h) in (2) 
and replacing Equation (3) by Equation (5). An alternative modification would be to keep 
the unified approach criterion -R&(A, n) but calibrate the associated tests differently, by 
replacing Pb+\ with q^ x in Equation (3) and, therefore, c(A) with c n (A) (except that R not 

R is used). We have explored this approach and found it to be very similar to the one 
presented. It has the disadvantage that the limits for n = are slightly dependent on b. 
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Our approach may be contrasted with that of Giunti, who has suggested a different 
modification of the unified approach, called the new ordering approach. His physical argu- 
ments are along similar lines to ours. However, in detail his approach differs. In the new 
ordering approach, Rb(X,n) is replaced by R^°(X,n) = p\+b( n ) /P\ NO +b( n ) m Equation 3, 
where X NO is the Bayes' estimate of A for a uniform prior. (We shall describe the Bayes' 
approach further in the next section.) The calibration then proceeds as in Equation 3, 
using Pb+\(k). The resulting intervals are shorter than ours, but depend on b when n = 0. 
Amusingly, our intervals are closer to the Bayesian intervals than are Giunti' s intervals, 
even though our approach is entirely frequentist. See Table 1 below. 



In Equation (4), 



n , \ Pb+x(n) 
%x(n) = -p^y (6) 



is the conditional probability of n events (total) given at most n background events. This 
is a very intuitive quantity but, unfortunately, is not a density in n, since -Pft(n) < 1 
for all n and, therefore, YlnLoQbxi 71 ) > Yl^=oPb+x( n ) = 1- Of course, Q^(n) could be 
renormalized by k(X) := J2nLo lb >( n )' an< ^ ^ ne resu hi n g ratio q^ x (n)/K,(X) would be a 
density; but using the ratio in a model would implicitly change the likelihood function. 
The density then lacks the intuitive appeal of q£ x (k), since the definition of the experiment 
producing this density then becomes unclear. 
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A closely related quantity is the conditional probability of at most n events total, given 
at most n background events 

Pb+\(n) 



D b ,x(n) = 



Pb(n) 



(7) 



It is not obvious, but D^^n) is a distribution function in n for reasons explained below. Let 
db,\(n) = Db,\( n ) ~ Db,\( n — 1) denote the corresponding density. Still another alternative 
is to replace Pb+x by d^x in the unified approach. This too led to a procedure that was 
more complicated and no more efficient than the modification described in Section 2. 

To see that D^ xin) is a distribution function in n, first observe that lim,^^ D^ xin) = 
lim n ^oo Pb + x(n) / Pb(n) = 1/1 = 1. So, it suffices to show that Db^x(n) is non-decreasing 
in n. For this, note that, after some manipulation, db ; x(n) can be written in either of the 
following forms for n > 0: 



db,x(n) = d obj x(n) - 



= d obj x(n) 



Pb(n)/T!k=oPb( k ) 
P\+b{n)/YljZlp\+b{j) 



(8), 



where 



dob,x(n) 



Pb+x(n) 

YTj=QPb{3) 



(9). 



Db,x{ n ) will be a non-decreasing function of n if the correction term in the second expression 
above is always < 1. Using the fact that these are Poisson distributions, 



Pb(n)/H 



~iPb(k) 



~ b b n /nl Epo^ (b+A) (fr + A)Vi! 



Px + b(n)/EUP^b(j) Ek= e-W/kl e-(W-*)(6 + A)»/n 



< i. 



(10) 



The last inequality occurs since b + A > b. 
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4. The Bayesian Connection 

The discussion in this section makes use of the following identity, which may be estab- 
lished by repeated integrations by parts: if m is any positive integer and c > 0, then 

p y (m)dy = — f y m e^dy = £ 1 C V C = P c (m), (11) 

This has an amusing consequence: While q^\{n) is not a density in n, it is a density in A; 
that is, 

00 

Jql x {n)d\=l. (12) 


It follows that qb,\(n) is the (formal) posterior distribution that is obtained when A is given 
an (improper) uniform distribution over the interval < A < 00. (It is also the limiting 
posterior that is obtained if A is given a (proper) uniform distribution over the interval 
< A < A and then A is allowed to approach 00). Moreover, using (11) again, leads to 
the following curious relation 

J ql\ x (n)dX = D bM (n). (13) 
Ao 

That is, the posterior probability that A exceeds Ao given n is the conditional probability 
of at most n events total given at most n background events when A = Ao- Hence, 
using D, one of our possibilities above, although fully based on a frequentist approach, 
has some Bayesian justification. Equation (13) also provides frequentist justification for 
conditioning. For it follows from (13) and Theorem 3.3 of Hwang et.al.® that D^^n) is 
an admissible p- value for testing Hq : A > Ao- Admissibility of the unconditional p- value 
P\ (n) is unclear to us at this writing, if b > 0. 

The Giunti approach, mentioned above, fundamentally uses a partly Bayesian, partly 
frequentist approach. 

Treating q bX (n) as the posterior density in A leads to Bayesian credible (confidence) 
intervals of the form {A : q^x( n ) — c "}? where c n is so chosen to control the posterior 
probability of coverage; that is, 

J q% x (n)d\ = 1 - a. (14) 

{A:g" A (n)>c„} 

Relation (13) is useful in computing the latter integral. The endpoints of these intervals 
have been computed for selected b and n and are compared to the endpoints of the modified 
unified approach in the table below. 
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Table 1. Comparison of Confidence levels for the unified, modified unified, Bayesian, 
and new ordering approaches described here, for 6 = 3. 





Unified 


Modified 


Bayesian 


New Ord. 


n (observed) 


Lower 


Upper 


Lower 


Upper 


Lower 


Upper 


Lower 


Upper 





0.0 


1.08 


0.0 


2.42 


0.0 


2.30 


0.0 


1.86 


f 


0.0 


1.88 


0.0 


2.94 


0.0 


2.84 


0.0 


2.49 


2 


0.0 


3.04 


0.0 


3.74 


0.0 


3.52 


0.0 


3.60 


3 


0.0 


4.42 


0.0 


4.78 


0.0 


4.36 


0.0 


4.86 


4 


0.0 


5.60 


0.0 


6.00 


0.0 


5.34 


0.0 


5.80 


5 


0.0 


6.99 


0.0 


7.26 


0.0 


6.44 


0.0 


7.21 


6 


0.f5 


8.47 


0.42 


8.40 


0.0 


7.60 


0.28 


8.65 


7 


0.89 


9.53 


0.96 


9.56 


0.55 


9.18 


1.02 


9.68 


8 


1.51 


11.0 


1.52 


11.0 


1.20 


10.59 


1.78 


11.2 


9 


1.88 


12.3 


1.88 


12.22 


1.90 


11.91 


2.49 


12.4 


fO 


2.63 


13.5 


2.64 


13.46 


2.63 


13.19 


3.10 


13.7 



5. Summary 

We have suggested a modification to the unified approach of Feldman and Cousins to 
further improve the estimation of signal counts in the presence of background. It consists 
of replacing the density function corresponding to the Poisson distribution Pb+\(k), with 
the conditional density function qg x (k). We noted that this method has a clear frequentist 
justification and is the answer to a clear statistics question. 

We compared the results using this modification to the unified approach with the 
results obtained using the unmodified unified approach. In contradistinction to the old 
method, the new method leads naturally to sensible results if the observation has fewer 
events than expected from background events alone. 
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